Fine-grained Visual Categorization Research Articles

Semi-Supervised Learning for FGVC With Out-of-Category Data.

Despite great strides made on fine-grained visual classification (FGVC), current methods are still heavily reliant on fully-supervised paradigms where ample expert labels are called for. Semi-supervised learning (SSL) techniques, acquiring knowledge from unlabeled data, provide a considerable means forward and have shown great promise for coarse-grained problems. However, exiting SSL paradigms mostly assume in-category (i.e., category-aligned) unlabeled data, which hinders their effectiveness when re-proposed on FGVC. In this paper, we put forward a novel design specifically aimed at making out-of-category data work for semi-supervised FGVC. We work off an important assumption that all fine-grained categories naturally follow a hierarchical structure (e.g., the phylogenetic tree of "Aves" that covers all bird species). It follows that, instead of operating on individual samples, we can instead predict sample relations within this tree structure as the optimization goal of SSL. Beyond this, we further introduced two strategies uniquely brought by these tree structures to achieve inter-sample consistency regularization and reliable pseudo-relation. Our experimental results reveal that (i) the proposed method yields good robustness against out-of-category data, and (ii) it can be equipped with prior arts, boosting their performance thus yielding state-of-the-art results.

Study on Fine-Grained Visual Classification of Low-Resolution Urinary Erythrocyte.

The morphological analysis test item of urine red blood cells is referred to as "extracorporeal renal biopsy," which holds significant importance for medical department testing. However, the accuracy of existing urine red blood cell morphology analyzers is suboptimal, and they are not widely utilized in medical examinations. Challenges include low image spatial resolution, blurred distinguishing features between cells, difficulty in fine-grained feature extraction, and insufficient data volume. This article aims to improve the classification accuracy of low-resolution urine red blood cells. This paper proposes a super-resolution method based on category-aware loss and an RBC-MIX data enhancement approach. It optimizes the cross-entropy loss to maximize the classification boundary and improve intra-class tightness and inter-class difference, achieving fine-grained classification of low-resolution urine red blood cells. Experimental outcomes demonstrate that with this method, an accuracy rate of 97.8% can be achieved for low-resolution urine red blood cell images. This algorithm attains outstanding classification performance for low-resolution urine red blood cells with only category labels required. This method can serve as a practical reference for urine red blood cell morphology examination items.

Dual-Dependency Attention Transformer for Fine-Grained Visual Classification.

Visual transformers (ViTs) are widely used in various visual tasks, such as fine-grained visual classification (FGVC). However, the self-attention mechanism, which is the core module of visual transformers, leads to quadratic computational and memory complexity. The sparse-attention and local-attention approaches currently used by most researchers are not suitable for FGVC tasks. These tasks require dense feature extraction and global dependency modeling. To address this challenge, we propose a dual-dependency attention transformer model. It decouples global token interactions into two paths. The first is a position-dependency attention pathway based on the intersection of two types of grouped attention. The second is a semantic dependency attention pathway based on dynamic central aggregation. This approach enhances the high-quality semantic modeling of discriminative cues while reducing the computational cost to linear computational complexity. In addition, we develop discriminative enhancement strategies. These strategies increase the sensitivity of high-confidence discriminative cue tracking with a knowledge-based representation approach. Experiments on three datasets, NABIRDS, CUB, and DOGS, show that the method is suitable for fine-grained image classification. It finds a balance between computational cost and performance.

Discriminative Suprasphere Embedding for Fine-Grained Visual Categorization.

Despite the great success of the existing work in fine-grained visual categorization (FGVC), there are still several unsolved challenges, e.g., poor interpretation and vagueness contribution. To circumvent this drawback, motivated by the hypersphere embedding method, we propose a discriminative suprasphere embedding (DSE) framework, which can provide intuitive geometric interpretation and effectively extract discriminative features. Specifically, DSE consists of three modules. The first module is a suprasphere embedding (SE) block, which learns discriminative information by emphasizing weight and phase. The second module is a phase activation map (PAM) used to analyze the contribution of local descriptors to the suprasphere feature representation, which uniformly highlights the object region and exhibits remarkable object localization capability. The last module is a class contribution map (CCM), which quantitatively analyzes the network classification decision and provides insight into the domain knowledge about classified objects. Comprehensive experiments on three benchmark datasets demonstrate the effectiveness of our proposed method in comparison with state-of-the-art methods.

Delving into Multimodal Prompting for Fine-Grained Visual Classification

Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category, which poses challenges due to subtle inter-class discrepancies and large intra-class variations. However, prevailing approaches primarily focus on uni-modal visual concepts. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks, yet the applicability of such models to FGVC tasks remains uncertain. In this paper, we aim to fully exploit the capabilities of cross-modal description to tackle FGVC tasks and propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image pertaining (CLIP) model. Our MP-FGVC comprises a multimodal prompts scheme and a multimodal adaptation scheme. The former includes Subcategory-specific Vision Prompt (SsVP) and Discrepancy-aware Text Prompt (DaTP), which explicitly highlights the subcategory-specific discrepancies from the perspectives of both vision and language. The latter aligns the vision and text prompting elements in a common semantic space, facilitating cross-modal collaborative reasoning through a Vision-Language Fusion Module (VLFM) for further improvement on FGVC. Moreover, we tailor a two-stage optimization strategy for MP-FGVC to fully leverage the pre-trained CLIP model and expedite efficient adaptation for FGVC. Extensive experiments conducted on four FGVC datasets demonstrate the effectiveness of our MP-FGVC.

Roll with the Punches: Expansion and Shrinkage of Soft Label Selection for Semi-supervised Fine-Grained Learning

While semi-supervised learning (SSL) has yielded promising results, the more realistic SSL scenario remains to be explored, in which the unlabeled data exhibits extremely high recognition difficulty, e.g., fine-grained visual classification in the context of SSL (SS-FGVC). The increased recognition difficulty on fine-grained unlabeled data spells disaster for pseudo-labeling accuracy, resulting in poor performance of the SSL model. To tackle this challenge, we propose Soft Label Selection with Confidence-Aware Clustering based on Class Transition Tracking (SoC) by reconstructing the pseudo-label selection process by jointly optimizing Expansion Objective and Shrinkage Objective, which is based on a soft label manner. Respectively, the former objective encourages soft labels to absorb more candidate classes to ensure the attendance of ground-truth class, while the latter encourages soft labels to reject more noisy classes, which is theoretically proved to be equivalent to entropy minimization. In comparisons with various state-of-the-art methods, our approach demonstrates its superior performance in SS-FGVC. Checkpoints and source code are available at https://github.com/NJUyued/SoC4SS-FGVC.

ViTree: Single-Path Neural Tree for Step-Wise Interpretable Fine-Grained Visual Categorization

As computer vision continues to advance and finds widespread applications across various domains, the need for interpretability in deep learning models becomes paramount. Existing methods often resort to post-hoc techniques or prototypes to explain the decision-making process, which can be indirect and lack intrinsic illustration. In this research, we introduce ViTree, a novel approach for fine-grained visual categorization that combines the popular vision transformer as a feature extraction backbone with neural decision trees. By traversing the tree paths, ViTree effectively selects patches from transformer-processed features to highlight informative local regions, thereby refining representations in a step-wise manner. Unlike previous tree-based models that rely on soft distributions or ensembles of paths, ViTree selects a single tree path, offering a clearer and simpler decision-making process. This patch and path selectivity enhances model interpretability of ViTree, enabling better insights into the model's inner workings. Remarkably, extensive experimentation validates that this streamlined approach surpasses various strong competitors and achieves state-of-the-art performance while maintaining exceptional interpretability which is proved by multi-perspective methods. Code can be found at https://github.com/SJTU-DeepVisionLab/ViTree.

R2-trans: Fine-grained visual categorization with redundancy reduction

Fine-grained visual categorization (FGVC) aims to discriminate similar subcategories, whose main challenge is the large intraclass diversities and subtle inter-class differences. Existing FGVC methods usually select discriminant regions found by a trained model, which is prone to neglect other potential discriminant information. On the other hand, the massive interactions between the sequence of image patches in ViT make the resulting class token contain lots of redundant information, which may also impact FGVC performance. In this paper, we present a novel approach for FGVC, which can simultaneously make use of partial yet sufficient discriminative information in environmental cues and also compress the redundant information in class-token with respect to the target. Specifically, our model calculates the ratio of high-weight regions in a batch, adaptively adjusts the masking threshold, and achieves moderate extraction of background information in the input space. Moreover, we also use the Information Bottleneck (IB) approach to guide our network to learn a minimum sufficient representations in the feature space. Experimental results on three widely-used benchmark datasets verify that our approach can achieve better performance than other state-of-the-art approaches and baseline models. The code of our model is available at: https://github.com/SYe-hub/R-2-Trans.

Multi-directional guidance network for fine-grained visual classification

Multi-directional guidance network for fine-grained visual classification

FET-FGVC: Feature-enhanced transformer for fine-grained visual classification

The challenge of Fine-grained visual classification (FGVC) comes from the small variations between classes and the large variations within classes. Inspired by the fact that identifying bird species focuses not only on the global features of the subject area but also on the subtle details of the local area, we propose a feature-enhanced Transformer to improve the performance of FGVC. Our proposed method consists of a Dynamic Swin Transformer backbone for extracting comprehensive global image features through continuous attention aggregation, a GCN-based local branch for separating and enhancing local features in different regions, and a pairwise feature interaction (PFI) module for enhancing global features through interactions between image pairs. We conducted extensive experiments on five FGVC datasets to demonstrate the superiority of our method. By fusing the enhanced global and local features, our method achieves the best accuracy compared to existing methods. Our method has an advantage in terms of computational efficiency.

The Image Data and Backbone in Weakly Supervised Fine-Grained Visual Categorization: A Revisit and Further Thinking

Weakly-supervised fine-grained visual categorization (FGVC) aims to achieve subclass classification within the same large class using only label information. Compared to general images, fine-grained images have similar appearances and features, and are often affected by disturbances such as viewpoint, lighting, and occlusion during data collection, resulting in significant intra-class variance and small inter-class variance. To achieve FGVC, carefully designed models are often needed to explore the locally discriminative regions of the image. This paper revisits high-quality FGVC publications based on deep learning and analyzes from two new perspective: fine-grained image data and backbone. We address two ignored but interesting problems in FGVC. First, we argue that the reasons for exacerbating intra-class variance are not the same in data of animal, plant, and commodity types, and it is necessary to consider the effects of posture, covariate shift, and structural changes. Additionally, the “soft boundary” between subclasses intensifies the difficulty of classification. Second, we highlight that convolutional networks and self-attention networks have different receptive fields and shape biases, leading to performance differences when processing different types of fine-grained data. Overall, our analysis provides new insights into recent advances, challenges, and future directions for FGVC based on deep learning, which can help researchers develop more effective models for FGVC.

Penalizing the Hard Example But Not Too Much: A Strong Baseline for Fine-Grained Visual Classification.

Though significant progress has been achieved on fine-grained visual classification (FGVC), severe overfitting still hinders model generalization. A recent study shows that hard samples in the training set can be easily fit, but most existing FGVC methods fail to classify some hard examples in the test set. The reason is that the model overfits those hard examples in the training set, but does not learn to generalize to unseen examples in the test set. In this article, we propose a moderate hard example modulation (MHEM) strategy to properly modulate the hard examples. MHEM encourages the model to not overfit hard examples and offers better generalization and discrimination. First, we introduce three conditions and formulate a general form of a modulated loss function. Second, we instantiate the loss function and provide a strong baseline for FGVC, where the performance of a naive backbone can be boosted and be comparable with recent methods. Moreover, we demonstrate that our baseline can be readily incorporated into the existing methods and empower these methods to be more discriminative. Equipped with our strong baseline, we achieve consistent improvements on three typical FGVC datasets, i.e., CUB-200-2011, Stanford Cars, and FGVC-Aircraft. We hope the idea of moderate hard example modulation will inspire future research work toward more effective fine-grained visual recognition.

Hierarchical Self-Distilled Feature Learning for Fine-Grained Visual Categorization.

Fine-grained visual categorization (FGVC) relies on hierarchical features extracted by deep convolutional neural networks (CNNs) to recognize closely alike objects. Particularly, shallow layer features containing rich spatial details are vital for specifying subtle differences between objects but are usually inadequately optimized due to gradient vanishing during backpropagation. In this article, hierarchical self-distillation (HSD) is introduced to generate well-optimized CNNs features for accurate fine-grained categorization. HSD inherits from the widely applied deep supervision and implements multiple intermediate losses for reinforced gradients. Besides that, we observe that the hard (one-hot) labels adopted for intermediate supervision hurt the performance of FGVC by enforcing overstrict supervision. As a solution, HSD seeks self-distillation where soft predictions generated by deeper layers of the network are hierarchically exploited to supervise shallow parts. Moreover, self-information entropy loss (SIELoss) is designed in HSD to adaptively soften intermediate predictions and facilitate better convergence. In addition, the gradient detached fusion (GDF) module is incorporated to produce an ensemble result with multiscale features via effective feature fusion. Extensive experiments on four challenging fine-grained datasets show that, with neglectable parameter increase, the proposed HSD framework and the GDF module both bring significant performance gains over different backbones, which also achieves state-of-the-art classification performance.

Fine-grained Lesion Classification Framework for Early Auxiliary Diagnosis.

The deep neural networks are envisaged for the early disease diagnosis from medical images. However, in the early stage of the disease, the medical images of patients and healthy people have only subtle visual differences. Distinguishing the medical images for early diagnosis belongs to the Fine-Grained Visual Classification (FGVC) task. Many recent works are based on a standard FGVC learning paradigm: locate the discriminative regions first and then classify by fusing the information of these regions. However, it is still not enough for medical images. Because the shape and size of the lesions are variable, and the relationship between lesions and the background is complex. In order to solve these problems, we propose a fine-grained lesion classification framework for early auxiliary diagnosis. We first locate and extract multiple lesions with different sizes and shapes from the original image and then fuse the feature of lesion and background based on attention mechanism. As shown by experiment results in two real-world clinical data sets, our model can locate accurately and perform better.

TransIFC: Invariant Cues-aware Feature Concentration Learning for Efficient Fine-grained Bird Image Classification

Fine-grained bird image classification (FBIC) is not only meaningful for endangered bird observation and protection but also a prevalent task for image classification in multimedia processing and computer vision. However, FBIC suffers from several challenges, such as bird molting, complex background, and arbitrary bird posture. To effectively tackle these challenges, we present a novel invariant cues-aware feature concentration Transformer (TransIFC), which learns invariant and core information in bird images. To this end, two novel modules are proposed to leverage the characteristics of bird images, namely, the hierarchy stage feature aggregation (HSFA) module and the feature in feature abstraction (FFA) module. The HSFA module aggregates the multiscale information of bird images by concatenating multilayer features. The FFA module extracts the invariant cues of birds through feature selection based on discrimination scores. Transformer is employed as the backbone to reveal the long-dependent semantic relationships in bird images. Moreover, abundant visualizations are provided to prove the interpretability of the HSFA and FFA modules in TransIFC. Comprehensive experiments demonstrate that TransIFC can achieve state-of-the-art performance on the CUB-200-2011 dataset (91.0%) and the NABirds dataset (90.9%). Finally, extended experiments have been conducted on the Stanford Cars dataset to suggest the potential of generalizing our method on other fine-grained visual classification tasks.

Neighbor self-knowledge distillation

Self-Knowledge Distillation (Self-KD), a technique that enables neural networks to learn from themselves, often relies on auxiliary modules or networks to generate supervisory signals for training. However, this approach incurs significant additional resource costs. Moreover, incorporating auxiliary classifiers within the network architecture creates a capacity mismatch when distilling knowledge from deep to shallow classifiers. This paper proposes a concise and efficient Self-KD method called Neighbor Self-Knowledge Distillation (NSKD), which introduces teacher assistants into the Self-KD by adding auxiliary classifiers to the shallow part of the network to construct distillations of multiple neighboring student-teacher assistant combinations to reduce the mismatch between the students' and teachers' abilities. During distillation, NSKD utilizes only the soft labels generated by each classifier and corresponding ground truth labels as supervisory signals, minimizing resource consumption. NSKD enables neighboring modules to learn from each other through neighboring distillation, enhancing overall network performance. Experimental results on five network models and seven popular datasets demonstrate the superiority of NSKD over other state-of-the-art Self-KD methods. Notably, NSKD achieves average accuracy improvements of 2.26%, 2.32%, and 2.4% on the CIFAR100, TinyImageNet, and fine-grained visual classification datasets.

Performacnes comparison of CNN-based models for fine-grained image classification

Fine-grained visual classification is regarded as a more refined classification that can identify specific types of objects. It has been widely used in commodity sales, vehicle recognition and person recognition, etc., and has played a great value in many fields. For example, it requires an algorithm to identify different species of birds or dogs to facilitate more practical applications. This task is difficult since the objects has similar appearance and there exist obvious intra-class variance and limited inter-class differences, where different kinds of birds could share very similar appearance. Deep learning techniques has been applied to image recognition, natural language processing, and many other fields. Several approaches tackling the fine-grained classification problem are proposed. To further demonstrate the different designs of these solutions, in this paper, fine-grained identification methods are compared and analyzed, among which WS-DAN achieves better results and it is preferred to be an effective method, which is expected to be more widely used in this field.

Cervical cancer classification based on a bilinear convolutional neural network approach and random projection

Cervical cancer (CC) is one of the most prevalent malignancies affecting women globally, with a particularly notable impact and high mortality in regions with limited economic resources. This underscores the imperative for the expeditious development of techniques that facilitate timely and precise detection, thereby augmenting treatment efficacy, enhancing survival rates, and mitigating the burden of healthcare expenditure. The intricacies of cervical cancer detection are inherently aligned with the challenges of fine-grained visual classification. This study focuses on the integration of bilinear pooling within convolutional neural networks (CNNs) and addresses the problem of the computational complexity of bilinear features with the fortification of the bilinear CNN with a random projection paradigm (RP-BCNN). The aim is the simultaneous achievement of improved classification precision and streamlined processing temporalities. The proposed methodology entails the introduction of a dyadic feature extraction protocol in which the input cellular image is subjected to twin feature extraction modalities. The feature maps obtained from this process undergo element-wise multiplication via an outer-product operation, thereby engendering composite feature representations. Subsequently, a judiciously designed random projection procedure is invoked to reduce dimensionality, yielding a more succinct yet informative image descriptor. Empirical evaluations of the introduced model predicted on the RP-BCNN framework yielded commendable outcomes. Notably, an accuracy of 0.9983 was achieved for the dual-label classification scenarios, and an accuracy of 0.9530 was realized in the context of multiclass classification encompassing seven distinct labels. The proposed model achieves an optimal equilibrium between classification accuracy and processing efficiency, thus constituting a potent instrument for the classification of cervical cancer. Further, it holds promise for the refinement of diagnostic accuracy, thereby providing a vantage point for embracing sophisticated techniques in the realm of medical image analysis.

Two-stage fine-grained image classification model based on multi-granularity feature fusion

Fine-grained visual classification (FGVC) is a difficult task due to the challenges of discriminative feature learning. Most existing methods directly use the final output of the network which always contains the global feature with high-level semantic information. However, the differences between fine-grained images are reflected in subtle local regions which often appear in the front of the network. When the texture of the background and object are similar or the proportion of the background is too large, the prediction will be greatly affected. In order to solve the above problems, this paper proposes multi-granularity feature fusion module (MGFF) and two-stage classification based on Vision-Transformer (ViT). The former comprehensively represents images by fusing features of different granularities, thus avoiding the limitations of single-scale features. The latter leverages the ViT model to separate the object from the background at a very small cost, thereby improving the accuracy of the prediction. We conduct comprehensive experiments and achieves the best performance in two fine-grained tasks on CUB-200-2011 and NA-Birds.

Coping with change: Learning invariant and minimum sufficient representations for fine-grained visual categorization

Fine-grained visual categorization (FGVC) is a challenging task due to similar visual appearances between various species. Previous studies always implicitly assume that the training and test data have the same underlying distributions, and that features extracted by modern backbone architectures remain discriminative and generalize well to unseen test data. However, we empirically justify that these conditions are not always true on benchmark datasets. To this end, we combine the merits of invariant risk minimization (IRM) and information bottleneck (IB) principle to learn invariant and minimum sufficient (IMS) representations for FGVC, such that the overall model can always discover the most succinct and consistent fine-grained features. We apply the matrix-based Rényi’s α-order entropy to simplify and stabilize the training of IB; we also design a “soft” environment partition scheme to make IRM applicable to FGVC task. To the best of our knowledge, we are the first to address the problem of FGVC from a generalization perspective and develop a new information-theoretic solution accordingly. Extensive experiments demonstrate the consistent performance gain offered by our IMS. Code is available at: https://github.com/SYe-hub/IMS.